Method for optically measuring three-dimensional coordinates and calibration of a three-dimensional measuring device

ABSTRACT

A method for scanning and obtaining three-dimensional (3D)l coordinates is provided. The method includes providing a 3D measuring device having a projector, a first camera and a second camera. The method records images of a light pattern emitted by the projector onto an object. A deviation in a measured parameter from an expected parameter is determined. The calibration of the 3D measuring device may be changed when the deviation is outside of a predetermined threshold.

CROSS REFERENCE TO RELATED APPLICATIONS (IF APPLICABLE)

The present application claims the benefit of German Patent Application10 2014 019 669.0 filed on Dec. 30, 2014, German Patent Application 102014 019 670.4 filed on Dec. 30, 2014, German Patent Application 10 2014019 672.0 filed on Dec. 30, 2014 and German Patent Application 10 2014019 671.2 filed on Dec. 30, 2014. The contents of all of which areincorporated by reference herein in their entirety.

The present application also a continuation-in-part application of U.S.application Ser. No. 14/826,859 filed on Aug. 14, 2015. U.S. Ser. No.14/826,859 claims the benefit of German Patent Application 10 2014 113015.4 filed on Sep. 10, 2014 and is also a continuation-in-partapplication of U.S. application Ser. No. 14/712,993 filed on May 15,2015, which is a nonprovisional application of U.S. ProvisionalApplication 62/161,461 filed on May 14, 2015. U.S. application Ser. No.14/712,993 further claims the benefit of German Patent Application 102014 013 677.9 filed on Sep. 10, 2014. The contents of all of which areincorporated by reference herein in their entirety.

The present application is further a continuation-in-part application ofU.S. application Ser. No. 14/722,219 filed on May 27, 2015, which is anonprovisional application of the aforementioned U.S. ProvisionalApplication 62/161,461. U.S. Ser. No. 14/722,219 claims the benefit ofGerman Application 10 2014 013 678.7 filed on Sep. 10, 2014. Thecontents of all of which are incorporated by reference herein in theirentirety.

BACKGROUND OF THE INVENTION

The subject matter disclosed herein relates to a portable scanner, andin particular to a portable scanner having a display.

A portable scanner includes a projector that projects light patterns onthe surface of an object to be scanned. The position of the projector isdetermined by means of a projected, encoded pattern. Two (or more)cameras, the relative positions and alignment of which are known or aredetermined, can record images of the surface with a further, uncodedpattern. The three-dimensional coordinates (of the points of thepattern) can be determined by means of mathematical methods which areknown per se, such as epipolar geometry.

In video gaming applications, scanners are known as tracking devices, inwhich a projector projects an encoded light pattern onto the target tobe pursued, preferably the user who is playing, in order to then recordthis encoded light pattern with a camera and to determine thecoordinates of the user. The data are represented on an appropriatedisplay.

A system for scanning a scene, including distance measuring, mayinclude, in its most simplest form, a camera unit with two cameras, andillumination unit and a synchronizing unit. The cameras, which mayoptionally include filters, are used for the stereoscopic registrationof a target area. The illumination unit is used for generating a patternin the target area, such as by means of a diffractive optical element.The synchronizing unit synchronizes the illumination unit and the cameraunit. Camera unit and illumination unit can be set up in selectablerelative positions. Optionally, also two camera units or twoillumination units can be used.

BRIEF DESCRIPTION OF THE INVENTION

According to one aspect of the invention, a method for opticallyscanning and measuring an environment is provided. The method comprisingthe steps of: providing a three-dimensional (3D) measurement devicehaving a first camera, a second camera and a projector, the 3Dmeasurement device further having a control and evaluation deviceoperably coupled to the at least one camera and the projector, thecontrol and evaluation device having memory; emitting a light patternonto an object with the projector; recording a first image of the lightpattern with the first camera at a first time; recording a second imageof the light pattern with the second camera at the first time; producinga 3D scan of the object based at least in part on the first image andthe second image; selecting at least one point in the first image andthe second image; determining a first 3D coordinates of the at least onepoint in the first image based at least in part on a first set ofcalibration parameters; determining a second 3D coordinates of the atleast on point in the second image based at least in part on the firstset of calibration parameters; determining a deviation based at least inpart on the first 3D coordinates and the second 3D coordinates;determining a second set of calibration parameters based at least inpart on the deviation; and storing the second set of calibrationparameters in memory.

According to another aspect of the invention, another method foroptically scanning and measuring an environment is provided. The methodcomprising: providing a three-dimensional (3D) measurement device havinga first camera, a second camera and a projector, the 3D measurementdevice further having a control and evaluation device operably coupledto the at least one camera and the projector, the control and evaluationdevice having memory; emitting at a first wavelength a light patternonto an object with the projector; recording a first image of the lightpattern with the first camera at a first time; recording a second imageof the light pattern with the second camera at the first time; producinga 3D scan of the object based at least in part on the first image andthe second image; selecting a first point in the center of the lightpattern, the first point being in the first image and the second image;selecting a second point in the light pattern, the second point beingpositioned away from the second of the light pattern; determining adeviation in the first wavelength based on the first point and thesecond point; and storing the deviation in memory.

These and other advantages and features will become more apparent fromthe following description taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWING

The subject matter, which is regarded as the invention, is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 shows a perspective view of a hand-held scanner and of an objectin the environment;

FIG. 2 shows a view of the front side of the hand-held scanner;

FIG. 3 shows a view of the reverse side of the hand-held scanner;

FIG. 4 shows a top view of the hand-held scanner from above;

FIG. 5 shows a view of the hand-held scanner from the right side;

FIG. 6 shows a perspective view corresponding to FIG. 1 without housing;

FIG. 7 shows a representation of the control and evaluation device withdisplay;

FIG. 8 shows a representation of the control and evaluation device ofFIG. 7 from a right side;

FIG. 9 shows the fields of view of the cameras with shaded overlap;

FIG. 10 shows the geometrical relationship of the image planes,projector plane, and epipolar lines;

FIG. 11 shows an exemplary method of autocalibration;

FIG. 12 shows a simplified situation of an inconsistency;

FIG. 13 shows a possible error field of the situation of FIG. 12;

FIG. 14 an example with one epipolar line;

FIG. 15 an example with two epipolar lines;

FIG. 16 an example with inconsistencies;

FIG. 17 a first verification of several epipolar lines;

FIG. 18 a second verification of several epipolar lines;

FIG. 19 shows a verification of the relative geometry of the cameras;

FIG. 20 shows a verification of the wavelength; and

FIG. 21 shows a typical error field of FIG. 20.

The detailed description explains embodiments of the invention, togetherwith advantages and features, by way of example with reference to thedrawings.

DETAILED DESCRIPTION OF THE INVENTION

In one embodiment, the carrying structure is stable mechanically andthermally, defines the relative distances and the relative alignments ofa camera and of a projector. The arrangement on a front side of the 3Dmeasuring device faces on the environment, has the advantage that thesedistances and alignments are not changed by a change of the shape of ahousing.

The term “projector” is defined to generally refer to a device forproducing a pattern. The generation of the pattern can take place bymeans of deflecting methods, such as generation by means of diffractiveoptical elements or micro-lenses (or single lasers), or by shadingmethods, for example the production by means of shutters, transparencies(as they would be used in a transparency projector) and other masks. Thedeflecting methods have the advantage of less light getting lost andconsequently a higher intensity being available.

Depending on the number of assemblies provided for distance measuring, acorresponding number of arms of the carrying structure is provided,which protrude from a common center located at the intersection of thearms. The assemblies, which may include a combination of cameras andprojectors, are provided in the area of the ends of the assigned arms.The assemblies may be arranged each on the reverse side of the carryingstructure. Their respective optics are directed through an assignedaperture in the carrying structure, so that the respective assembliesare operably oriented to face towards the environment from the frontside. A housing covers the reverse side and forms the handle part.

In one embodiment, the carrying structure consists of acarbon-reinforced or a glass-fiber-reinforced matrix of syntheticmaterial or ceramics (or of another material). The material provides forstability and a low weight and can, at the same time, be configured withviewing areas. A concave (spherical) curvature of the front side of thecarrying structure does not only have constructive advantages, but italso protects the optical components arranged on the front side when thefront surface of the 3D measuring device is placed on a work surface.

The projector produces the projected pattern, which may or may not bewithin the visible wavelength range. In one embodiment, the projectedpattern has a wavelength in the infrared range. The two cameras areconfigured to acquire images from light within this wavelength range,while also filtering out scattered light and other interferences in thevisible wavelength range. A color or 2D camera can be provided as thirdcamera for additional information, such as color for example. Suchcamera records images of the environment and of the object beingscanned. In an embodiment where the camera captures color, the pointcloud generated from the scanning process (herein referred to as the“3D-scan”) can have color values assigned from the color informationcontained in the color images.

During operation the 3D measuring device generates multiple 3D scans ofthe same scene, from different positions. The 3D scans are registered ina joint coordinate system. For joining two overlapping 3D scans, thereare advantages in being able to recognizable structures within the 3Dscans. Preferably, such recognizable structures are looked for anddisplayed continuously or, at least after the recording process. If, ina determined area, density is not at a desired level, further 3D scansof this area can be generated. A subdivision of the display used forrepresenting a video image and the (thereto adjacent parts of the)three-dimensional point cloud helps to recognize, in which areas scanshould still be generated.

In one embodiment, the 3D measuring device is designed as a portablescanner, i.e. it works at high speed and is of a size and weightsuitable for carrying and use by a single person. It is, however, alsopossible to mount the 3D measuring device on a tripod (or on anotherstand), on a manually movable trolley (or another cart), or on anautonomously moving robot, i.e. that it is not carried by theuser—optionally also by using another housing, for example without acarrying handle. It should be appreciated that while embodiments hereindescribe the 3D measuring device as being hand-held, this is forexemplary purposes and the claimed invention should not be so limited.In other embodiments, the 3D measuring device may also be configured asa compact unit, which are stationary or mobile and, if appropriate,built together with other devices.

Referring to FIGS. 1-6, a 3D measuring device 100 is provided asportable part of a device for optically scanning and measuring anenvironment of the 3D measuring device 100 with objects O. As usedherein, the side of the device 100 which faces the user shall bereferred to as the reverse side, and the side of the device 100 whichfaces the environment as the front side. This definition extends to thecomponents of the 3D measuring device 100. The 3D measuring device 100is provided (on its front side) visibly with a carrying structure 102having three arms 102 a, 102 b, 102 c. These arms give the carryingstructure 102 a T-shape or a Y-shape, i.e. a triangular arrangement. Thearea in which the three arms 102 a, 102 b, 102 c intersect and areconnected with each other, and from which the three arms 102 a, 102 b,102 c protrude, defines the center of the 3D measuring device 100. Fromthe user's view, the carrying structure 102 is provided with a left arm102 a, a right arm 102 b and a lower arm 102 c. In one embodiment, theangle between the left arm 102 a and the right arm 102 b is, forexample, approximately 150°+20°, between the left arm 102 a and thelower arm 102 c approximately 105°+10°. The lower arm 102 c is, in someembodiments, somewhat longer than the two other arms 102 a, 102 b.

The carrying structure 102 preferably is configured fromfiber-reinforced synthetic material, such as a carbon-fiber-reinforcedsynthetic material (CFC). In another embodiment, the carrying structure102 is made from carbon-fiber-reinforced ceramics or fromglass-fiber-reinforced synthetic material. The material renders thecarrying structure 102 mechanically and thermally stable and provides atthe same time for a low weight. The thickness of the carrying structure102 is considerably smaller (for example 5 to 15 mm) than the length ofthe arms 102 a, 102 b, 102 c (for example 15 to 25 cm). The carryingstructure 102 hence has a flat basic shape. In some embodiments, thearms 102 a, 102 b, 102 c, may include a reinforced back near the centerof the arm. It is, however, preferably not configured to be plane, butto be curved. Such curvature of the carrying structure 102 is adapted tothe curvature of a sphere having a radius of approximately 1 to 3 m. Thefront side (facing the object O) of the carrying structure 102 isthereby configured to be concave, the reverse side to be convex. Thecurved shape of the carrying structure 102 is advantageous for providingstability. The front side of the carrying structure 102 (and in oneembodiment the visible areas of the reverse side) is configured to be aviewing area, i.e. it is not provided with hiders, covers, cladding orother kinds of packaging. The preferred configuration fromfiber-reinforced synthetic materials or ceramics is particularlysuitable for this purpose.

On the reverse side of the carrying structure 102, a housing 104 isarranged, which is connected with the carrying structure 102 within thearea of the ends of the three arms 102 a, 102 b, 102 c in a floatingway, by means of appropriate connecting means, for example by means ofrubber rings and screws with a bit of clearance. As used herein, afloating connection is one that reduces or eliminates the transmissionof vibration from the housing 104 to the carrying structure 102. In oneembodiment, the floating connection is formed by a rubber isolationmount disposed between the housing 104 and the carrying structure. Inone embodiment, an elastomeric seal, such as rubber, is disposed betweenthe outer perimeter of the carrying structure 102 and the housing 104.The carrying structure 102 and the housing 104 are then clamped togetherusing elastomeric bushings. The seal and bushings cooperate to form thefloating connection between the carrying structure 102 and the housing104. Within the area of the left arm 102 a and of the right arm 102 b,the edge of the housing 104 extends into the immediate vicinity of thecarrying structure 102, while the housing 104 extends from the center ofthe 3D measuring device 100 within the area of the lower arm 102 c, at adistance to the carrying structure 102, forming a handle part 104 g,bends off at the end of the handle part 104 g and approaches the end ofthe lower arm 102 c, where it is connected with it in a floating manner.The edge of the handle 104 g extends into the immediate vicinity of thecarrying structure 102. In some embodiments, sections of the carryingstructure 102 may include a reinforced back 102 r. The back 102 rprotrudes into the interior of the housing 104. The housing 104 acts asa hood to cover the reverse side of the carrying structure 102 anddefine an interior space.

The protective elements 105 may be attached to the housing 104 or to thecarrying structure 102. In one embodiment, the protective elements 105are arranged at the ends of and extend outward from the arms 102 a, 102b, 102 c to protect the 3D measuring device from impacts and from damageresulting thereof. When not in use, the 3D measuring device 100 can beput down with its front side to the bottom. Due to the concave curvatureof the front side, on the 3D measuring device will only contact thesurface at the ends of the arms 102 a, 102 b, 102 c. In embodimentswhere the protective elements 105 are positioned at the ends of the arms102 a, 102 b, 102 c advantages are gained since the protective elements105 will provide additional clearance with the surface. Furthermore,when the protective elements 105 are made from a soft material forexample from rubber, this provides a desirable tactile feel for theuser's hand. This soft material can optionally be attached to thehousing 104, particularly to the handle part 104 g.

On the reverse side of the 3D measuring device 100, an control actuatoror control knob 106 is arranged on the housing 104, by means of which atleast optical scanning and measuring, i.e. the scanning process, can bestarted and stopped. The control knob 106 is arranged in the center ofthe housing 104 adjacent one end of the handle. The control knob 106 maybe multi-functional and provide different functions based on a sequenceof actions by the user. These actions may time based (e.g. multiplebutton pushed within a predetermined time), or space based (e.g. thebutton moved in a predetermined set of directions), or a combination ofboth. In one embodiment, the control knob 106 may be tilted in severaldirections in (e.g. left, right, up, down). In one embodiment, aroundthe control knob 106 there are at least one status lamp 107. In oneembodiment, there may be a plurality of status lamps 107. These statuslamps 107 may be used to show the actual status of the 3D measuringdevice 100 and thus facilitate the operation thereof. The status lamps107 can preferably show different colors (for example green or red) inorder to distinguish several status'. The status lamps 107 may be lightemitting diodes (LEDs).

On the carrying structure 102, spaced apart from each other at a defineddistance, a first camera 111 is arranged on the left arm 102 a (in thearea of its end), and a second camera 112 is arranged on the right arm102 b (in the area of its end). The two cameras 111 and 112 are arrangedon the reverse side of the carrying structure 102 and fixed thereto,wherein the carrying structure 102 is provided with apertures throughwhich the respective camera 111, 112 can acquire images through thefront side of the carrying structure 102. The two cameras 111, 112 arepreferably surrounded by the connecting means for the floatingconnection of the housing 104 with the carrying structure 102.

Each of the cameras 111, 112 have a field of view associated therewith.The alignments of the first camera 111 and of the second camera 112 toeach other are adjusted or adjustable in such a way that the fields ofview overlap to allow stereoscopic images of the objects O. If thealignments are fixed, there is a desired predetermined overlappingrange, depending on the application in which the 3D measuring device 100is used. Depending on environment situations, also a range of severaldecimeters or meters may be desired. In another embodiment, thealignments of the cameras 111, 112 can be adjusted by the user, forexample by pivoting the cameras 111, 112 in opposite directions. In oneembodiment, the alignment of the cameras 111, 112 is tracked andtherefore known to the 3D measuring device 100. In another embodiment,the alignment is initially at random (and unknown), and is thendetermined, such as be measuring the positions of the camera's forexample, and thus known to the 3D measuring device 100. In still anotherembodiment, the alignment is set and fixed during manufacturing orcalibration of the 3D measurement device 100. In embodiments where thefirst and second cameras 111, 112 are adjusted, a calibration may beperformed in the field to determine the angles and positions of thecameras in the 3D measuring device 100. The types of calibrations thatmay be used are discussed further herein.

The first camera 111 and the second camera 112 are preferablymonochrome, i.e. sensitive to a narrow wavelength range, for example bybeing provided with corresponding filters, which then filter out otherwavelength ranges, including scattered light. This narrow wavelengthrange may also be within the infrared range. In order to obtain colorinformation on the objects O, the 3D measuring device 100 preferablyincludes a 2D camera, such as color camera 113 which is preferablyaligned symmetrically to the first camera 111 and to the second camera112, and arranged in the center of the 3D measuring device 100, betweenthe cameras 111, 112. The 2D camera 113 may include an image sensor thatis sensitive to light in the visible wavelength range. The 2D camera 113captures 2D images of the scene, i.e. the environment of the 3Dmeasuring device 100, including the objects O therein included.

In order to illuminate the scene for the 2D camera, in the event ofunfavorable lighting conditions, at least one, in the illustratedembodiment a light source, such as four (powerful) light-emitting diodes(LED) 114 are provided. One radiating element 115 is associated witheach of the LEDs 114. The light emitted from the light-emitting diode114 is deflected in correspondence with the alignment of the 3Dmeasuring device 100, from the corresponding LED 114. Such a radiatingelement 115 can, for example, be a lens or an appropriately configuredend of a light guide. The (in the illustrated embodiment four) radiatingelements 115 are arranged equally around the color camera 113. Each LED114 is connected with the assigned radiating element 115 by means of onelight guide each. The LED 114 therefore can be structurally arranged ata control unit 118 of the 3D measuring device 100, such as by beingfixed on a board thereof.

In order to later have a reference for the images recorded by thecameras 111,112, 113, a sensor such as an inclinometer 119 is provided.In one embodiment, the inclinometer 119 is an acceleration sensor (withone or several sensitive axes), which is manufactured in a manner knownper se, as MEMS (micro-electro-mechanical system). As inclinometer 119,also other embodiments and combinations are possible. The data of the 3Dmeasuring device 100 each have (as one component) a gravitationdirection provided by the inclinometer 119.

During operation images are recorded by the first camera 111 and by thesecond camera 112. From these images three-dimensional data can bedetermined, i.e. 3D-scans of the objects O can be produced, for exampleby means of photogrammetry. The objects O, however, may have fewstructures or features and many smooth surfaces. As a result, thegeneration of 3D-scans from the scattered light of the objects O isdifficult.

To resolve this difficulty, a projector 121 may be used, which isarranged at the lower arm 102 c (in the area of its end). The projector121 is arranged within the interior space on the reverse side of thecarrying structure 102 and fixed thereto. The carrying structure 102 isprovided with an aperture through which the projector 121 can project apattern of light through the front side of the carrying structure 102.In one embodiment, the projector 121 is surrounded by the connectingmeans to provide a floating connection between the housing 104 with thecarrying structure 102. The projector 121, the first camera 111, and thesecond camera 112 are arranged in a triangular arrangement with respectto each other and aligned to the environment of the 3D measuring device100. The projector 121 is aligned in correspondence with the two cameras111, 112. The relative alignment between the cameras 111, 112 and theprojector 121 is preset or can be set by the user.

In one embodiment, the cameras 111, 112 and the projector 121 form anequilateral triangle and have a common tilt angle. When arranged in thismanner, and if the field of view of the cameras 111, 112 and theprojector 121 are similar, the centers of the field of view willintersect at a common point at a particular distance from the scanner100. This arrangement allows for a maximum amount of overlap to beobtained. In embodiments where the tilt or angle of the cameras 111, 112and projector 121 may be adjusted, the distance or range to theintersection of the fields of view may be changed.

If the user places 3D measuring device 100 on a surface on its frontside, i.e. with the front side to the surface, the concave curvature ofthe front side creates a gap between the cameras 111, 112, 113 and theprojector 121 from the surface, so that the respective lenses areprotected from damage.

The cameras 111, 112, 113, the projector 121, the control knob 106, thestatus lamps 107, the light-emitting diodes 114 and the inclinometer 119are connected with the common control unit 118, which is arranged insidethe housing 104. This control unit 118 can be part of a control andevaluation device which is integrated in the housing. In an embodiment,the control unit 118 is connected with a standardized communicationinterface at the housing 104, the interface being configured for awireless connection (for example Bluetooth, WLAN, DECT) as an emittingand receiving unit, or for a cable connection (for example USB, LAN), ifappropriate also as a defined interface, such as that described in DE 102009 010 465 B3, the contents of which are incorporated by referenceherein. The communication interface is connected with an externalcontrol and evaluation device 122 (as a further component of the devicefor optically scanning and measuring an environment of the 3D measuringdevice 100), by means of said wireless connection or connection bycable. In the present case, the communication interface is configuredfor a connection by cable, wherein a cable 125 is plugged into thehousing 104, for example at the lower end of the handle part 104 g, sothat the cable 125 extends in prolongation of the handle part 104 g.

The control and evaluation device 122 may include one or more processors122 a having memory. The processor 122 a being configured to carry outthe methods for operating and controlling the 3D measuring device 100and evaluating and storing the measured data. The control and evaluationdevice 122 may be a portable computer (notebook) or a tablet (orsmartphone) such as that shown in FIGS. 7 and 8, or any external ordistal computer (e.g. in the web). The control and evaluation device 122may also be configured in software for controlling the 3D measuringdevice 100 and for evaluating the measured data. However, the controland evaluation device 122 may be embodied in separate hardware, or itcan be integrated into the 3D measuring device 100. The control andevaluation device 122 may also be a system of distributed components, atleast one component integrated into the 3D measuring device 100 and onecomponent externally. Accordingly, the processor(s) 122 a for performingsaid methods may be embedded in the 3D measuring device 100 and/or in anexternal computer.

The projector 121 projects a pattern X, which it produces, for exampleby means of a diffractive optical element, on the objects O to bescanned. The pattern X does not need to be encoded (that is to saysingle-valued), but may be uncoded, for example periodically (i.e.multivalued). The multi-valuedness is resolved by the use of the twocameras 111, 112, combined with the available, exact knowledge of theshape and direction of the pattern. The uncoded pattern may, forexample, be a projection of identical periodically spaced patternelements (e.g. spots or lines of light). The correspondence between theprojected pattern elements from the projector 121 and the patternelements in the images on the photosensitive arrays of the cameras 111,112 is determined through simultaneous epipolar constraints and usingcalibration parameters, as discussed further herein. The uniqueness inthe correspondence of the points is achieved by the use of the twocameras 111 and 112, combined with the knowledge of the shape anddirection of the pattern, the combined knowledge being obtained from acalibration of the 3D measuring device 100.

As used herein, the term “pattern element” should means the shape of anelement of the pattern X, while the term “point” should indicate theposition (of a pattern element or of something else) in 3D coordinates.

The uncoded pattern X (FIG. 1) is preferably a point pattern, comprisinga regular arrangement of points in a grid. In the present invention, forexample, approximately one hundred times one hundred points (10,000points total) are projected over a field of view FOV (FIG. 9) at anangle of approximately 50° to a distance of approx. 0.5 m to 5 m. Thepattern X can also be a line pattern or a combined pattern of points andlines, each of which is formed by tightly arranged light points. Thefirst camera 111 comprises a first image plane B111, and the secondcamera 112 comprises a second image plane B112. The two cameras 111 and112 receive at least a portion of the pattern X in their respectiveimage planes B111 and B112, in which the photosensitive arrays (forexample CMOS or CCD arrays) are arranged to capture a portion of thepattern X reflected from the object O.

There is a relationship between the point density, the distance betweenthe projector 121 and the object O and the resolution that can beobtained with the produced pattern X. With diffractive patterngeneration, the light of one source is distributed over the pattern. Inthat case the brightness of the pattern elements depends on the numberof elements in the pattern when the total power of the light source islimited. Depending on the intensity of the light scattered from theobjects and the intensity of background light it may be determinedwhether it is desirable to have fewer but brighter pattern elements.Fewer pattern elements means the acquired point density decreases. Ittherefore seems helpful to be able to generate, in addition to patternX, at least one other pattern. Depending on the generation of thepatterns, a dynamic transition between the patterns and/or a spatialintermingling is possible, in order to use the desired pattern for thecurrent situation. In an embodiment, the projector 121 may produce thetwo patterns offset to each other with respect to time or in anotherwavelength range or with different intensity. The other pattern may be apattern which deviates from pattern X, such as an uncoded pattern. Inthe illustrated embodiment the pattern is a point pattern with a regulararrangement of points having another distance (grid length) to eachother.

In one embodiment, the pattern X is a monochromatic pattern. The patternX may be produced by means of a diffractive optical element 124 in theprojector 121. The diffractive optical element 124 converts a singlebeam of light from a light source 121 a in FIG. 20 to a collection ofcollection of beams, each having lower optical power than the singlebeam. Each of the collection of beams traveling in a different directionto produce a spot when striking the object O. The light source 121 a maybe a laser, a superluminescent diode, or an LED, for example. In anembodiment, the wavelength of the light source 121 a is in the infraredrange. The lateral resolution is then limited only by the diameter andspacing of the spots of light in the projected pattern X. If the patternX is in the infrared range, it is possible to capture the images of theobject O and surrounding environment with the 2D camera 113 withoutinterference from the pattern X. Similarly, if the pattern X is producedin the ultraviolet light wavelength range, the images acquired by the 2Dcamera 113 would not have interference from the pattern X.

In one embodiment, the projector 121 produces the pattern X on theobjects O only during the time periods when the cameras 111, 112 (and ifavailable 113) are recording images of the objects O. This providesadvantages in energy efficiency and eye protection. The two cameras 111,112 and the projector 121 are synchronized or coordinated with eachother, with regard to both, time and the pattern X used. Each recordingprocess starts by the projector 121 producing the pattern X, similar toa flash in photography, followed by the cameras 111, 112 (and, ifavailable 113) acquiring pairs of images, in other words one image eachfrom each of the two cameras 111, 112. As used herein, these pairs ofimages that are acquired at substantially the same time are referred toas “frames.” The recording process can comprise one single frame (shot),or a sequence of a plurality of frames (video). Such a shot or such avideo is triggered by means of the control knob 106. After processing ofthe data, each frame then constitutes a 3D-scan consisting of a pointcloud in the three-dimensional space. This point cloud is defined in therelative local coordinate system of the 3D measuring device 100.

The data furnished by the 3D measuring device 100 are processed in thecontrol and evaluation device 122 to generate the 3D scans from theframes. The 3D scans in turn are joined or registered in a jointcoordinate system. For registering, the known methods can be used, suchas by identifying natural or artificial targets (i.e. recognizablestructures) in overlapping areas of two 3D scans. Through identificationof these targets, the assignment of the two 3D scans may be determinedby means of corresponding pairs. A whole scene (a plurality of 3D scans)is thus gradually registered by the 3D measuring device 100. The controland evaluation device 122 is provided with a display 130 (displaydevice), which is integrated or connected externally.

In the exemplary embodiment, the projector 121 is not collinear with thetwo cameras 111 and 112, but rather the camera's 111, 112 and projector121 are arranged to form a triangle. As shown in FIG. 10, thistriangular configuration enables the use of epipolar geometry based onmathematic methods of optics. The constraints of epipolar geometryprovide that a point on the projector plane P₁₂₁ of the projector 121falls on a first epipolar line on the first image plane B₁₁₁ and on asecond epipolar line of the second image plane B₁₁₂, the epipolar linesfor each of the image planes B₁₁₁ and B₁₁₂ being determined by therelative geometry of the projector 121 and the two cameras 111 and 112.Further, a point on the first image plane B₁₁₁ falls on an epipolar lineof the projector plane P₁₂₁ and on an epipolar line of the second imageplane B₁₁₂, the epipolar lines in the projector plane and second imageplane being determined by the relative geometry of the projector 121 andcameras 111, 112. Further still, a point on the second image plane B₁₁₂falls on an epipolar line of the projector plane P₁₂₁ and on an epipolarline of the first image plane B₁₁₁, the epipolar lines in the projectorplane and the first image plane being determined by the relativegeometry of the projector 121 and cameras 111, 112. It can be seen thatthe use of at least two cameras and one projector provides sufficientepipolar constraints to enable a correspondence among points in thepattern X to be determined for the points on the image planes B₁₁₁ andB₁₁₂ and the projector plane P₁₂₁, even though the projected patternelements have no distinguishing characteristics, such as having anidentical shape for example (an uncoded pattern).

In one embodiment, at least three units (e.g. projector 121 and the twocameras 111, 112) are used to generate the 3D scenes. This allows forunambiguous triangular relations of points and epipolar lines from whichthe correspondence of projections of the pattern (X) in the two imageplanes B₁₁₁, B₁₁₂ can be determined. Due to the additional stereogeometry relative to a pair of cameras, considerably more of the pointsof the pattern, which otherwise cannot be distinguished, can beidentified on an epipolar line “e.” The density of features on theobject O can thus simultaneously be high, and the size of the pattern Xfeature (e.g. the spot) can be kept very low. This contrasts with othermethods that utilize encoded patterns where the size of the feature inthe pattern has a lower limit based on the resolution of the projector,this size limitation in coded patterns limits the lateral resolution ofthe 3D scan. Once the correspondence among the points X on the projector121 and cameras 111, 112 has been determined, the three-dimensionalcoordinates of the points on the surface of the object O may bedetermined for the 3D-scan data by means of triangulation.

Triangulation calculations may be performed between the two cameras 111,112 based on the baseline distance between the two cameras 111, 112 andthe relative angles of tilt of the two cameras 111, 112. Triangulationcalculations may also be performed between the projector 121 and firstcamera 111 and between the projector 121 and the second camera 112. Toperform these triangulation calculations, a baseline distance is needsto be determined between the projector 121 and the first camera 111 andanother baseline distance is needs to be determined between theprojector 121 and the second camera 112. In addition, the relativeangles of tilt between the projector/first camera and projector/secondcamera is used.

In principle, any one of the three triangulation calculations issufficient to determine 3D coordinates of the points X on the object O,and so the extra two triangulation relations provides redundantinformation (redundancies) that may be usefully employed to provideself-checking of measurement results and to provide self-calibrationfunctionality as described further herein below. As used herein, theterm “redundancy” refers to multiple determinations of 3D coordinatesfor a particular point or set of points on the object.

Additional three-dimensional data can be gained by means ofphotogrammetry methods by using several frames with different camerapositions, for example from the 2D camera 113 or from a part of an imageacquired by the cameras 111, 112. To perform photogrammetry calculationsthe objects viewed by the cameras 111, 112, 113 should be illuminated.Such illumination may be background illumination, such as from the sunor artificial lights for example. The background illumination may beprovided by the 3D measuring device 100 or by another external lightsource. In an embodiment, the object is illuminated with light from LEDs114. Illumination enables the two-dimensional cameras 111, 112, 113 todiscern properties of the object such as color, contrast, and shadow,which facilitate identification of object features.

The measuring process may also have a temporal aspect. Typically,uncoded patterns were used with stationary devices to allow an entiresequence of patterns to be projected and images be captured in order todetermine a single 3D-scan. In order for this 3D scan to be determined,both the scanner and the object needed to remain stationary relative toeach other. In contrast, embodiments of the present invention generateone 3D-scan for each set of images acquired by the cameras 111, 112. Inanother embodiment (not shown), a second projector is arranged adjacentto the present projector 121 or a further diffractive optical element isprovided. This second projector emits at least one second pattern on tothe object O in addition to pattern X. In an embodiment having twoprojectors, it is possible to switch between the pattern X and thesecond pattern to capture with one set of images with the differentpatterns consecutively. Thus, the 3D-scan has a higher resolution bycombining the evaluation results obtained from the different patterns.

In order to determine the coordinates of the entire object, the 3D-scanswhich are produced from the images acquired by the cameras 111, 112 needto be registered. In other words, the three-dimensional point clouds ofeach frame must be inserted into a common coordinate system.Registration is possible, for example, by videogrammetry, such as“structure from motion” (SFM) or “simultaneous localization and mapping”(SLAM) methods for example. The natural texture of the objects O mayalso be used for common points of reference. In one embodiment, astationary pattern is projected and used for registration.

The data furnished by the 3D measuring device 100 is processed in thecontrol and evaluation device 122 by generating 3D scans from the imageframes. The 3D scans in turn are then joined or registered in a commoncoordinate system. As discussed above, registration methods know in theart may be used, such as by using natural or artificial targets (i.e.recognizable structures) for example. These targets can be localized andidentified in order to determine the assignment or alignment of twodifferent 3D scans relative to each other by means of correspondingpairs. The 3D scans (sometimes colloquially referred to as a “scene”)may then be gradually registered by the 3D measuring device 100. In theexemplary embodiment, the control and evaluation device 122 is providedwith a display 130 (display device) to allow the user to review the 3Dscans.

An embodiment is shown in FIG. 11 for the process of the opticallyscanning and measuring the environment of the 3D measuring device 100.In a first process block 201, the capturing of a pair of images for oneframe are performed and the measured values are determined and recorded.In the exemplary embodiment, each image is a 2D array of valuesrepresenting the voltages corresponding to the number of electrons ineach pixel well.

The process then proceeds to a second process block 202, where themeasured values are evaluated and the 3D-scan data are generated fromthe frame. In the second process block, the numbers are evaluated incombination with the known positions of the projected spots from theprojector to determine the 3D coordinates of points on the object O. Thedistances for each of the pixels are obtained using the triangulationcalculation (including the epipolar geometry calculation) and the twoangles for each pixel are determined based on the location of the pixeland the camera geometry (lens focal length, pixel size, etc.).

The process then proceeds to a third process block 203 where multipleframes of 3D-scan data are registered in a common coordinate system. Inone embodiment, videogrammetry information is obtained with the centercolor camera 113 to assist in the registration. In one embodiment, arepresentation of the 3D scan data is displayed and stored. Finally, theprocess proceeds to a fourth process block 204 where an auto-calibrationis performed. In one embodiment, the auto-calibration performed infourth process block 204 may occur simultaneously with the secondprocess block 202 or may be performed sequentially therewith. As long asthe operator continues operation to additional frames are acquired, suchas by continuously pressing the control knob 106 for example, theprocess returns to the first process block 201.

In the second process block 202, images of particular points areselected in a frame for automatically determining correspondence. Theseselected points correspond to points on the object O, in particular topattern elements (e.g. spots) of the pattern X. For each image of aselected point in the first image plane B₁₁₁, the epipolar lines “e” areconsecutively located in the second image plane B₁₁₂ and in theprojector plane P₁₂₁. This procedure for locating the epipolar lines “e”is repeated for images of points in the second image plane B₁₁₂ and forimages of points in the projector plane P₁₂₁. These multiple epipolarconstraints enable the determination of one-to-one correspondence ineach of the three planes between the projected and received patternelements (e.g., identical spots). As shown in FIG. 10, the point X₀ (ofthe pattern X on the object O) is visible on all three planes B₁₁₁,B₁₁₂, P₁₂₁, and its image is on two epipolar lines “e” on each plane.The point X₀ is the intersection point of three straight lines, a lightbeam of the projector 121, and each one light beam of the two cameras111, 112. The point X₀ can thus be identified unambiguously, if thedensity of the points of the pattern X is sufficiently low.

Auto-calibration performed in block 204 takes place automatically anddoes not require the use of external calibration objects. Starting withthe initial calibration and later with the currently used calibration,the frames are examined for inconsistencies in terms of space(inconsistencies of geometry) and time (parameters change over time), inorder to correct calibration, if appropriate. The inconsistencies canhave thermal causes, such as due to an increase of operating temperaturefor example. The inconsistencies may also have mechanical causes, suchas a mechanical shock caused by the 3D measuring device 100 striking asurface or the floor for example. These inconsistencies may manifestthrough deviations in measured positions, angles and other geometricalfeatures of e.g. points on the objects or in the planes B₁₁₁, B₁₁₂,P₁₂₁.

The calibration parameters which may be corrected may be extrinsicparameters, intrinsic parameters, and operating parameters. Extrinsicparameters for each unit (cameras 111, 112, 113 and projector 121)include the six degrees of freedom of rigid bodies, i.e. three positionsand three angles. Particularly relevant is the relative geometry betweenthe cameras 111, 112, 113 and the projector 131, such as the relativedistances and relative angles of their alignments for example. Intrinsicparameters may be related to camera or projector device features such asbut not limited to the focal length, position of principal point,distortion parameters, centering of the photosensitive array or MEMSprojector array, scale of the array in each direction, rotation of thearray relative to the local coordinate system of the 3D measuring device100, and aberration correction coefficients for the camera or projectorlens system. Operating parameters may be the wavelength of the lightsource 121 a, the temperature and the humidity of the air.

With respect to FIG. 10, an inconsistency may be a deviation Δ of theactual position of the point X₀ with respect to its expected theoreticalposition in one of the three planes.

As explained herein above, epipolar constraints are solvedsimultaneously to determine the correspondence of projected and imagedpattern elements (i.e. their images) on the two cameras 111, 112 and oneprojector 121. Some redundant information (redundancies) from thesesimultaneous equations is available to reveal inconsistencies in thecorrespondences.

In addition, as explained herein above, three separate triangulationcalculations may be performed to obtain three sets of 3D coordinates.These three triangulation calculations are: 1) between the first andsecond cameras 111, 112 (stereo cameras); 2) between the projector 121and first camera 111; and 3) between the projector 121 and second camera112. The 3D coordinates obtained from the three different triangulationcalculations may be compared and, if inconsistencies are seen, changesmay be made to the calibration parameters. This provides a first methodto perform auto-calibration.

FIG. 12 shows a simplified situation of the inconsistency with two unitsU₁, U₂. The units U₁, U₂ may be either between the two cameras 111, 112or between the projector 121 and one of the cameras 111, 112 forexample. Each unit U₁, U₂ comprises a plane, in which points may beselected. The two epipolar lines “e” are common for both planes. Aselected point 236 in the plane of unit U₁ is corresponding to a point216 in the plane of unit U₂. It should be appreciated that both points216, 236 are the images of a real point on the object O. Thecorrespondence may be detected, such as when the point 216, 236 is theimage of a spot of the pattern X on the object O for example. In oneembodiment, the “spot” on the object O is illuminated and adjacent areais dark or darker than the illuminated point. However, when the distanceof the point 216, 236 perpendicular to the epipolar lines “e” is not thesame in both planes, but a deviation Δ occurs, i.e. a deviation Δbetween the position of the point 216 and an expected position 218.

In general, the deviation Δ is a vector. In an embodiment having twounits U₁, U2, such as the projector 121 and camera 111 for example, onlythe component of the deviation Δ perpendicular to the epipolar lines “e”is known. The component parallel to the epipolar lines “e” vanishes whendetermining the 3D coordinates. In an embodiment having more than twounits, such as the projector 121 and both cameras 111, 112 for example,the components of deviation Δ in both dimensions of the planes may bedetermined due to the redundant measurements (redundancies in detectingdeviations and in determining 3D coordinates). Having deviations Δ forseveral selected points, all determined deviations Δ may be drawn into amap shown in FIG. 13, the sometimes referred to as an error field. Incase of only two units, only one component of each deviation Δ can bedrawn into the error field. Where an inconsistency is due to a singlecause or reason, the error field is typical for a certain type ofinconsistency. FIG. 13 illustrates an example of an error field that isgenerated by the rotation of the first camera 111 about the direction ofview, such as when the calibration parameter for the roll angle of thefirst camera 111 is incorrect.

Referring now to FIGS. 14-18, the deviations Δ are now described withrespect to epipolar constraints. In general, there are two possibilitiesfor the involved units (cameras and projectors). One possibility is thatone of the units U₁, U₂ is a camera and the other is a projector. Theother possibility is that both units U₁, U₂ are cameras. It should beappreciated that the use of a single projector and two cameras is forexemplary purposes and the claimed invention should not be so limited.In other embodiments additional projectors or cameras may be used.Having three of more units (for example, having two cameras and oneprojector) gives additional capability in automatically determiningcalibration parameters, as discussed in more detail herein.

Each of the units U₁, U₂ has an origin point sometimes referred to asthe perspective center point O₁, O₂ or center of projection. This pointrepresents a point through which all rays pass out of the unit (for aprojector) or into the unit (for a camera). It should be appreciatedthat in an actual device, not all of the rays pass through theperspective center points, however corrections may be made in softwareto the calibration parameters of the camera system to bring thecorrected rays through these points. The two perspective center pointsO₁, O₂ define the baseline distance 208 between the units U₁, U₂.

Each of the units U₁, U₂ also has a plane 210, 230 in which images areformed. In a projector, this plane is known as the projector plane P₁₂₁and, in a camera, it is known as an image plane B₁₁₁, B₁₁₂. Typically,in an actual projector or camera, the projector plane P₁₂₁ or imageplane B₁₁₁, B₁₁₂ is behind the perspective center O₁, O₂ rather than infront of it as illustrated in FIG. 14. In most cases, a hardware devicesuch as a photosensitive array (in a camera) or a pattern generator (ina projector) is placed at the position of the plane behind theperspective center. However, it should be appreciated that the positionof the planes in front of the perspective centers as shown in FIG. 14 ismathematically equivalent to the planes being positioned on the otherside of the perspective centers.

The perspective centers O₁, O₂ are separated by a baseline distance B.The baseline 208 that connects the perspective centers O₁, O₂ intersectsthe planes 230, 210 in points E₁, E₂. The points of intersection arereferred to as epipolar points, also known as epipoles. A line drawnthrough one of the epipoles on a corresponding plane is referred to asan epipolar line. For the case of the plane 210 and correspondingepipole E₂, the epipolar line is 212. A point P₁ on the plane 230 lieson the epipolar line 212.

As explained above, each ray such as ray 232 passes through aperspective center point O₁ to arrive at a plane such as 230 in whichimages are formed. If the plane 230 is a projector plane, then the pointP₁ is projected onto an object at a point such as P_(A), P_(B), P_(C),or P_(D), depending on the distance to the object. These points P_(A),P_(B), P_(C), P_(D), which share the common ray 232, fall onto theepipolar line 212 at corresponding points Q_(A), Q_(B), Q_(C), or Q_(D)on unit U₂, which in this example is a camera. The ray 232 and theepipolar line 212 are both on the plane that includes the points O₁, O₂,and P_(D). If the plane 230 is a camera plane rather than a projectorplane, then the point P₁ received by the camera image plane may arrivefrom any point on the epipolar line 212, for example, from any of thepoints Q_(A), Q_(B), Q_(C), or Q_(D).

FIG. 15 shows an epipolar line 234 on the plane 230 of unit U₁ inaddition to the epipolar line 212 of the plane 210 of unit U₁. Any pointV₁ (and W_(A), W_(B), W_(C)) on the epipolar line 212 must have acorresponding point (image or projection point) U_(A), U_(B), U_(C),U_(D) on the epipolar line 234. Similarly, any point U_(A), U_(B),U_(C), U_(D) on the epipolar line 234 must have a corresponding pointW_(A), W_(B), W_(C), V₁ on the epipolar line 212. The set of points 240represent points, e.g. V_(A), V_(B), V_(C), V_(D), in space that mayintersect with an object O.

A 3D measuring device 100 is self-calibrating or auto-calibrating if, inthe routine performance of measurements by the 3D measuring device 100,the measurement results are also used to obtain corrected calibrationparameters (i.e. a corrected calibration). In an embodiment of thepresent invention, the first step to obtain corrected new calibrationparameters is to detect inconsistencies in the positions of images ofselected points on projection or image planes in relation to thepositions expected based on epipolar constraints.

An example of such an inconsistency is shown in FIG. 16. A point 236 onthe plane 230 intersects an object at the point 238. According toepipolar geometrical constraints, the point 238 should appear on theepipolar line 212 and specifically at the point 218. In this case, theactual point was observed at the position 216. In general, with only twoplanes 210, 230, all that is known is that, for a point 236 that fallson the epipolar line 234, a corresponding point should fall on theepipolar line 212. That the point 216 does not fall on the epipolar line212 indicates that there is a problem with the calibration parameters,although whether the calibration parameter(s) at fault is an extrinsicparameter, intrinsic parameter, or operating parameter is difficult tosay based on an observation of a single point 216.

FIG. 16 shows some errors that may be seen in extrinsic calibrationparameters. One type of error that may be seen is in the baseline 208,not only the baseline distance B, but also the specific position of theperspective centers O₁, O₂. In other words in the coordinates of theperspective center points along the direction of the nominal baseline208 (error 252) and in a direction perpendicular to the direction of thenominal baseline 208 (error 254). Another possible extrinsic error is inthe angular orientation of unit U₁ or unit U₂. One way to characterizethe orientation is in terms of a pitch angle 256 about an axis 255 and ayaw angle 258 about an axis 257. If the calibration parameters for thepitch and yaw angles of the planes 230 and 210 are incorrect, the pointson the planes will not correspond to the expected positions based onepipolar geometrical constraints. Unit U₁ and unit U₂ may also haveincorrect calibration parameters for the roll angle of the camera orprojector. Such errors may be detected and corrected in anauto-calibration procedure as discussed further herein. The roll anglecalibration parameter is sometimes considered to be an intrinsiccalibration parameter rather than an extrinsic parameter.

FIG. 17 and FIG. 18 illustrate how a pattern of points on a plane suchas planes 210, 230 can be used to identify and correct calibrationparameter errors. The rectangular region 300 represents a projector orimage plane such as the plane 210, 230. The solid lines 302 representepipolar lines that converge in an epipole K. The dashed lines 304, 306in FIG. 17 and FIG. 18 represent the apparent epipolar lines based onthe current calibration parameters. In FIG. 17, the measured pointsconverge to the epipole point K but the calibration parametersincorrectly indicate that the points converge to a point J. This type oferror might be caused by a misplacement of the baseline position of theunit, for example. In FIG. 18, the epipolar lines are rotated relativeto epipolar lines determined from the calibration parameters. Thisrotation 308 might result, for example, from an error in a calibrationparameter associated with the orientation of the unit U₁ or unit U₂, forexample, an error in the pitch angle, yaw angle, or roll angle.

If deviations are too large, difficulties in finding the correspondencesof a point X₀ might occur in simultaneously solving the epipolarconstraints. In an embodiment, the pattern X consists of a large number(e.g. 10,000) of spots with a low optical intensity and a smaller number(e.g. 1,000) of spots with a high optical intensity. With this variationin optical intensities, the 3D measuring device 100 may recognizeobjects having surfaces with high or low reflectance. If difficulties infinding the correspondences occur, it is possible to space the spotsprojected in the pattern X farther apart to reduce ambiguity indetermining correspondences during the fourth process block. In theembodiment having the variation in optical intensities, the spots withthe lower intensity, can be filtered out or at least be reduced byreducing the exposure times and/or reducing the total power of theprojector 121. In this instance, only the spots with high opticalintensity (which have a higher spacing) are visible on the cameras 111,112. This reduces the ambiguities in determining correspondences.

For fully and correctly determining the calibration parameters, it isadvantageous to use the whole volume around the 3D measuring device 100for measurements, in particular to use the depth information fordetermining the extrinsic parameters. As an example, FIG. 19 shows(schematically), how the relative geometry of the two cameras 111 and112 is verified. For this purpose, the two points X₁, X₂ are selectedwith a different distance to the 3D measuring device 100 (i.e. with adifferent depth). With each of the points X₁ or X₂ and the formercalibration, it can be verified whether the cameras 111 and 112 stillprovide consistent results. If an inconsistency occurs due to adeviation of the relative distance or of the relative alignment of thetwo cameras 111 and 112, it is possible to distinguish between the twokinds of error with the two different distances. Due to the highmechanical and thermal stability of the carrying structure 102, thermaldeformations of the 3D measuring device 100 or deformations thereofcaused by a mechanical shock would be unusual, but may occur at themounting means of the cameras 111, 112, 113 and of the projector 121 forexample. A verification of the calibration after switching initiatingoperation of the 3D measuring device 100 and after long intervals interms of time is sufficient in most cases, such as after the acquisitionof twenty to one hundred frames for example.

The calibration parameters may be amended according to the determineddeviations Δ, for example deviations detected by means of the selectedpoints X₀, X₁ or X₂ may be compensated. Using the redundancies inmeasuring the 3D coordinates of the points, new calibration parametersmay be determined. For example, the calibration parameters may bechanged (corrected) according to an optimization strategy until thedeviations Δ are minimized or are less than a predetermined threshold.The new calibration parameters are compared with the currently usedcalibration parameters and replace them, if appropriate, i.e. if thedeviation between the (corrected) new calibration parameters and thecurrently used calibration parameters exceeds a given threshold. Thusthe calibration is corrected. It should be appreciated that many of thedeviations due to mechanical issues are single events and may be fixedby a permanent correction of the calibration parameters, in particularthe extrinsic or intrinsic parameters.

Errors related to deviations in the operating parameters of theprojector 121 may be determined much faster than deviations in theextrinsic parameters or intrinsic parameters. In an embodiment, anoperating parameter to be checked is the wavelength of the light source121 a. The wavelength can change due to the warming up of the lightsource 121 a or changes in the pump current for example. An embodimentis shown in FIG. 20 (schematically). The pattern X generated by thediffractive element 124 changes in scale with the change in wavelength.As shown in the error field of FIG. 21, in the center of the pattern X,such as the position with the zeroth order of diffraction of the laserbeam, there is no deviation in the position of the central patternelement if the wavelength changes. The deviations Δ appear at locationswith higher orders of diffraction, such as in the more peripheralpattern elements for example, as a shift in position. Such a shift inposition of individual diffraction of the pattern elements can berecognized with a single camera 111 or 112.

If position and alignment of the 3D measuring device 100 (andconsequently of the projector 121) relative to the object O remainunchanged, a selected point on the object O moves from a point withtheoretical coordinates X₁ (i.e. the coordinates determined by means ofthe latest used calibration) in the 3D scan to a point with actualcoordinates X₂ which differ therefrom. In other words an expected pointX₁ becomes an actual point X₂. Using the depth information, as describedabove in relation to FIG. 19, the actual 3D coordinates of the selectedpoint X₂ may be determined from the frames of each single shot. In theevent of a deviation Δ of the actual coordinates of point X₂ from thetheoretical coordinates X₁ of the selected point, a change of thewavelength is detected. The new wavelength is determined from the sizeof the deviation Δ. In another embodiment, two frames captured atdifferent times are compared to detect a deviation of the actualcoordinates of the selected point X₂ from theoretical coordinates X₁.

If a change of the wavelength is observed, actions to counter the changecan be taken or calibration can be corrected, if appropriate. Correctionof calibration can be made by replacing the wavelength of thecalibration by the new wavelength detected. In other words, thecalibration parameter of the wavelength is changed. Actions to counterthe change may include, for example, cooling the laser (if the lightsource 121 a is a laser) or reducing the pump current, to return thewavelength to its unchanged (i.e. original) value. By applying feedbackin a control loop to the pump current or to the laser cooling, thewavelength of the laser can be stabilized.

Some light sources 121 a may drift in wavelength when no control systemis used. In these embodiments, broad optical bandpass filters might beused to cover the range of possible wavelengths. However, broad opticalfilters may pass more undesired ambient light, which may be an issuewhen working outdoors. Therefore in some applications a wavelengthcontrol system is desirable. It should be appreciated that theembodiments described herein of directly observing a change inwavelength provides advantages in simplifying a wavelength controlsystems, which ordinarily require stabilization of both temperature andcurrent.

Some types of semiconductor lasers such as Fabry Perot (FP) lasers emita single spatial mode but may support multiple longitudinal modes, whereeach longitudinal mode having a slightly different frequency. In somecases, current may be adjusted to switch between operation with singleor multiple longitudinal modes. The number of longitudinal modes of thelaser may be an operating parameter of the projector 121. Ordinarily,one single wavelength is desired as this produces a single well definedpattern X. If two wavelengths are produced by the laser, the secondwavelength produces a type of shadow in the desired pattern X. Theresult can be described with reference to FIG. 20. Depending on thedifference in the wavelengths of the multiple longitudinal modes, acircular point may become an elliptical or a barbell shape or a pointpair X₁, X₂. These deviations can be detected by applying imageprocessing to the received pattern X. If multiple modes are observed,pump current or cooling may be adjusted to reduce the number oflongitudinal modes to one.

Deviations in the intrinsic parameters of the cameras 111 and 112, forexample focal length, may in some cases be automatically recognized andbe compensated for. In other instances, deviations in intrinsicparameters may arise, not from thermal or mechanical causes, but due todeliberate actions by the operator or through the operation of the 3Dmeasuring device 100. These actions may include a change in the opticsof the cameras 111, 112, or by focusing or zooming (e.g. magnifying orincreasing scale) the cameras 111, 112 for example.

An additional possibility of verifying calibration results is to combinethe 3D scan data based on triangulation with 2D images captured by the2D camera 113. These 2D images normally comprise colors and otherfeatures of the object O. In one embodiment, the projector 121 and thetwo cameras 111 and 112 are synchronized with the camera 113 to captureframes of the object O synchronously. In this embodiment, the projector121 projects infrared light received by the cameras 111, 112 throughinfrared filters, while the camera 113 receives an image of the object Othough a filter that blocks infrared light wavelengths and thereforepattern X. Thus the camera 113 receives a clear image of the features ofthe object O. By comparing object features observed by the cameras 111,112 to those observed by the camera 113, errors in the calibrationparameters of the triangulation system that includes elements 111, 112,and 121 may be determined. The method used to determine errors in thecalibration parameters is similar to that described here, such as the 2Dcamera 113 being regarded as one of the units U₁, U₂ for example.Correspondences are detected in the captured frames and 3D-coordinatesof the selected object features are determined. Due to the redundanciesin determining the 3D coordinates, new calibration parameters may bedetermined, which are compared with the currently used calibrationparameters. If the difference between the current calibration parametersand the new calibration parameters exceeds a threshold, the newcalibration parameters replace the currently used ones.

In another embodiment, the light received by camera 113 is not filteredto exclude infrared light. Rather, in this embodiment, the camera 113capable of detecting and acquiring an image of the pattern X if presenton the object O. The camera 113 may then operate in either of two modes.In a first mode, the camera 113 collects images temporally in betweenpulsed projections of the pattern X by the projector 121. In this mode,the camera 113 captures images of features that may be compared to thefeatures calculated by the triangulation system that includes 111, 112,and 121 to determine errors in the calibration parameters. In a secondmode, the camera 113 acquires images when the pattern X is projectedonto the object O. In this case, errors in the relative alignment of thesix degrees-of-freedom of the camera 113 to the triangulation elements111, 112, and 121 may be determined.

For the case in which the images acquired by the 2D camera 113 arereceived between image frames of cameras 111, 112, an interpolation of3D and 2D coordinate data may be made to account for the interleaving ofimages. Such interpolations may be mathematical. In one embodiment theinterpolations may be based at least in part on a Kalman filtercalculation. In an embodiment, the 3D measuring device 100 is providedwith an inertial measurement unit (or another position tracking device).The inertial measurement unit measures accelerations in the threedirections of space with three sensors (accelerometers) and, with threefurther sensors (gyroscopes), the angular velocities around threecoordinate axes. Movements can thus be measured on short time scales.These movement measurements allow for a compensating offset in terms oftime between distance measured by the projector and cameras 111, 112 andthe image acquired by the 2D camera 113. This compensating offset allowsfor the combination of the data.

It should be appreciated that if deviations cannot be compensated for,such as due to their extent or to their frequent change for example,then no automatic correction of the latest calibration may be performed.In an embodiment, when the auto-calibration is not performed, thedeviations may be defined and displayed on the display 130 to alert theoperator of an issue with the calibration to further action may betaken. In the exemplary embodiment, the 2D camera 113 continuouslycaptures 2D images, which result in a current video live image VL (videolive image).

One embodiment of the display 130 shown in FIG. 7 illustrates asubdivided image or subdivided screen. In this embodiment, the display130 is divided into a first display part 130 a and a second display part130 b. In the present embodiment, the first display part 130 a is a(rectangular) central part of the display 130, and the second displaypart 130 b is a peripheral area around the first display part 130 a. Inanother embodiment, the two display parts may be columns. In theillustrated embodiment, the first display part 130 a is shown as havinga rectangular shape, however this is for exemplary purposes and theclaimed invention should not be so limited. In other embodiments, thefirst display part 130 a may have other shapes, including but notlimited to circular, square, trapezoid, trapezium, parallelogram, oval,triangular, or a polygon having any number of sides. In one embodiment,the shape of the first display part 130 a is user defined or selectable.

In the first display part 130 a the video live image VL is displayed,such as that captured by 2D camera 113 for example. In the seconddisplay part 130 b, an image of the latest 3D scan (or a plurality of 3Dscans that have been registered) is displayed as at least part of a viewof the three-dimensional point cloud 3DP. The size of the first displaypart 130 a may be variable, and the second display part 130 b isarranged in the area between the first display part 130 a and the border131 of the display 130. As video live image VL changes, such as when theuser moves the device 100, the image of the three-dimensional pointcloud 3DP changes correspondingly to reflect the change in position andorientation of the device 100.

It should be appreciated that the placement of the image of thethree-dimensional point cloud 3DP around the periphery of the video liveimage VL provides advantages in allowing the user to easily see whereadditional scanning may be required without taking their eyes off of thedisplay 130. In addition it may be desirable for the user to determineif the computational alignment of the current camera position to thealready acquired 3D data is within a desired specification. If thealignment is outside of specification, it would be noticed asdiscontinuities at the border between the image and thethree-dimensional point cloud 3DP.

The image acquired by the camera 113 is a two-dimensional (2D) image ofthe scene. A 2D image that is rendered into a three-dimensional viewwill typically include a pincushion-shaped or barrel-shaped distortiondepending on the type of optical lens used in the camera. Generally,where the field of view (FOV) of the camera 113 is small (e.g. about 40degrees), the distortion is not readily apparent to the user. Similarly,the image of the three-dimensional point cloud data may appear distorteddepending on how the image is processed for the display. The point clouddata 3DP may be viewed as a planar view where the image is obtained inthe native coordinate system of the scanner (e.g. a spherical coordinatesystem) and mapped onto a plane. In a planar view, straight lines appearto be curved. Further, the image near the center-top and center-bottomedges (e.g. the poles) may be distorted relative to a line extendingalong the midpoint of the image (e.g. the equator). Further, there mayalso be distortions created by trying to represent a spherical surfaceon a rectangular grid (similar to the Mercator projection problem).

It should be appreciated that it is desired to have the images withinthe first display part 130 a appear to be similar to that in the seconddisplay part 130 b to provide a continuous and seamless image experiencefor the user. If the image of three-dimensional point cloud 3DP issignificantly distorted, it may make it difficult for the user todetermine which areas could use additional scanning. Since the planarimage of the point cloud data 3DP could be distorted relative to the 2Dcamera image, one or more processing steps may be performed on the imagegenerated from the point cloud data 3DP. In one embodiment, the field ofview (FOV) of the second display part 130 b is limited so that only thecentral portion of the planar image is shown. In other words, the imageis truncated or cropped to remove the highly distorted portions of theimage. Where the FOV is small (e.g. less 120 degrees), the distortion islimited and the planar view of the point cloud data 3DP will appear asdesired to the user. In one embodiment, the planar view is processed toscale and shift the planar image to provide to match the camera 113image in the first display part 130 a.

In another embodiment, the three-dimensional point cloud data 3DP isprocessed to generate a panoramic image. As used herein, the termpanoramic refers to a display in which angular movement is possibleabout a point in space (generally the location of the user). A panoramicview does not incur the distortions at the poles as is the case with aplanar view. The panoramic view may be a spherical panorama thatincludes 360 degrees in the azimuth direction and +/−45 degrees ion thezenith. In one embodiment the spherical panoramic view may be only aportion of a sphere.

In another embodiment, the point cloud data 3DP may be processed togenerate a 3D display. A 3D display refers to a display in whichprovision is made to enable not only rotation about a fixed point, butalso translational movement from point to point in space. This providesadvantages in allowing the user to move about the environment andprovide a continuous and seamless display between the first display part130 a and the second display part 130 b.

In one embodiment, the video live image VL in the first display part 130a and the image of the three-dimensional point cloud 3DP in the seconddisplay part 130 b match together seamlessly and continuously (withrespect to the displayed contents). A part of the three-dimensionalpoint cloud 3DP is first selected (by the control and evaluation device122) in such a way, as it is regarded from the perspective of the 2Dcamera 113 or at least from a position aligned with the 2D camera 113.Then, the selected part of the three-dimensional point cloud 3DP isselected in such a way that it adjoins continuously the video live imageVL. In other words, the displayed image of the three-dimensional pointcloud 3DP becomes a continuation of the video live image VL for theareas beyond the field of view of the 2D camera 113 on the left, on theright, top and bottom relative to the field of view of the 2D camera).As discussed above, the selected portion of the three-dimensional pointcloud 3DP may be processed to reduce or eliminate distortions. In otherembodiments, the representation may correspond to the representation ofa fish-eye lens, but preferably it is undistorted. The part of thethree-dimensional point cloud 3DP which is located in the area occupiedby the first display part 130 a, in other words the portion beneath orhidden by the video live image VL, is not displayed.

It should be appreciated that the density of the points in thethree-dimensional point cloud 3DP in the area where the first displaypart 130 a is located will not be visible to the user. Normally, thevideo live image VL is displayed using the natural coloring. However, inorder to indicate the density of the points in the area covered/behindby the video live image VL, the coloring of the video live image VL maybe changed artificially such as by overlaying for example. In thisembodiment, the artificial color (and, if appropriate, the intensity)used for representing the artificially colored video live image VLcorresponds to the density of the points. For example, a green coloringto the video live image VL may indicate a (sufficiently) high densitywhile a yellow coloring may be used to indicate a medium or low pointdensity (e.g. areas which still the scan data can be improved). Inanother embodiment, the distant-depending precision of the data pointscould be displayed using this color-coding.

To support the registration of the 3D scans, flags or marks 133 (FIG. 7)may be inserted in the first display part 130 a to indicate structures(i.e. possible targets) recognized by the control and evaluation device122. The marks 133 may be a symbol, such as a small “x” or “+” forexample. The recognizable structures can be points, corners, edges ortextures of objects. The recognizable structures may be found by thelatest 3D scan or the video live image VL being subjected to thebeginning of the registering process (i.e. to the localization oftargets). The use of the latest video live image VL provides advantagesin that the registration process does not have to be performed asfrequently. If the marks 133 have a high density, it is considered to bea successful registration of the 3D scans. If, however, a lower densityof the marks 133 is recognized, additional 3D scans may be performedusing a relatively slow movement of the 3D measuring device 100. Byslowing the movement of the device 100 during the scan, additional orhigher density points may be acquired. Correspondingly, the density ofthe marks 133 may be used as a qualitative measure for the success ofthe registration. Similarly, the density of the points of thethree-dimensional point cloud 3DP may be used to indicate a successfulscan. As discussed above, the density of points in the scan may berepresented by the artificial coloring of the video live image VL.

The movement of the 3D measuring device 100 and processing of thecaptured frames may also be performed by a tracking function, i.e. the3D measuring device 100 tracks the relative movement of its environmentwith the methods used during tracking. If tracking gets lost, forexample, if the 3D measuring device 100 has been moved too fast, thereis a simple possibility of reassuming tracking. In this embodiment, thevideo live image VL as it is provided by the 2D camera 113 and the lastvideo still image from tracking provided by it may be representedadjacent to each other in a side by side arrangement on the display 130for the user. The user may then move the 3D measuring device 100 untilthe two video images coincide.

In one embodiment, the 3D measuring device 100 may be controlled basedon movements of the device 100. These movements or gestures by the usercan also be used for controlling the representation of the video imageVL or of the three-dimensional point cloud 3DP. In one embodiment, thescale of representation of the video image VL and/or of thethree-dimensional point cloud 3DP on the display 130 may depend on thespeed and/or acceleration of the movement of the 3D measuring device100. The term “scale” is defined as the ratio between the size (eitherlinear dimension or area) of the first display part 130 a and the sizeof the complete display 130, being denoted as a percentage.

A small field of view of the 2D camera 113 is assigned to a small scale.In the present embodiment with a subdivided display 130 with a centralfirst display part 130 a showing the video live image VL, this firstdisplay part 130 a then may be of smaller size than in the standardcase, and the second display part 130 b (about the periphery of thefirst display part 130 a) shows a bigger part of the three-dimensionalpoint cloud 3DP. A larger field of view is assigned to a large scale. Inone embodiment, the video live image VL may fill the whole display 130.

In the event of high speeds of movement of the 3D measuring device 100are detected, the scale of the representation may be configured smallerthan with low speeds and vice versa. Similarly, this may apply toaccelerations of the movement of the 3D measuring device 100. Forexample, the scale of the displayed image is reduced in the case ofpositive accelerations, and the scale is increased in the case ofnegative accelerations. The scale may also depend on a component of thespeed and/or acceleration of the movement of the 3D measuring device100, for example on a component which is arranged perpendicular orparallel to the alignment of the 3D measuring device 100. If the scaleis determined based on a component of the movement, parallel to thealignment (i.e. in the direction of the alignment), the scale can alsobe made dependent on the change of an average distance to objects O fromthe 3D measuring device 100.

In some embodiments, the change of the scale due to movement, astandstill of the movement of the 3D measuring device 100 or a thresholdspeed of movement value not being achieved can be used to record asequence of still images of the camera 113 with a low dynamic range.These images may be captured at low dynamic range but with differentexposure times or illumination intensities within the sequence togenerate a high dynamic range image therefrom.

In some embodiments, the direction of gravity may be defined at thebeginning of the registration process by a defined movement of the 3Dmeasuring device 100. This defined movement is carried out by the userby moving the device 100 in a vertical upward and downward movement forexample. In other embodiments, the direction of gravity may bedetermined from a set of statistics of all movements during theregistration process. A plane may be averaged from the coordinates ofthe positions taken by the device 100 while recording process along apath of movement through space. It is assumed that the averaged plane islocated horizontally in space, meaning that the direction of gravity isperpendicular to it. As a result, the use of inclinometer 119 fordetermining the direction of gravity may be avoided.

The evaluation of the coordinates of the positions may also be used fordetermining the kind of scene and, if appropriate, to offer differentrepresentations or operating possibilities. A path of movement around acenter location (with an alignment of the 3D measuring device 100oriented towards the interior), suggests an image of a single object O(object-centered image). Similarly, a path of movement that orients thedevice 100 towards the outside (and particularly longer straightsections of the path of movements) makes reference to an image of rooms.Thus, where it is determined that a room is being scanned, an image of afloor plan (top view) may be inserted into the display 130.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention mayinclude only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description, butis only limited by the scope of the appended claims.

1. Method for optically scanning and measuring an environment, themethod comprising: providing a three-dimensional (3D) measurement devicehaving a first camera, a second camera and a projector, the 3Dmeasurement device further having a control and evaluation deviceoperably coupled to the at least one camera and the projector, thecontrol and evaluation device having memory; emitting a light patternonto an object with the projector; recording a first image of the lightpattern with the first camera at a first time; recording a second imageof the light pattern with the second camera at the first time; producinga 3D scan of the object based at least in part on the first image andthe second image; selecting at least one point in the first image andthe second image; determining a first 3D coordinates of the at least onepoint in the first image based at least in part on a first set ofcalibration parameters; determining a second 3D coordinates of the atleast on point in the second image based at least in part on the firstset of calibration parameters; determining a deviation based at least inpart on the first 3D coordinates and the second 3D coordinates;determining a second set of calibration parameters based at least inpart on the deviation; and storing the second set of calibrationparameters in memory.
 2. The method of claim 1 further comprisingcomparing the deviation to a predetermined threshold and replacing thefirst set calibration parameters with the second set of calibrationparameters when the deviation exceeds the predetermined threshold. 3.The method of claim 1 wherein the step of determining the deviationincludes determining a difference in position between the first 3Dcoordinates and the second 3D coordinates.
 4. The method of claim 2wherein: the projector includes a projector plane; the first cameraincludes a first image plane; the second camera includes a second imageplane; the at least one point is selected to have correspondences on atleast two planes selected from the first image plane, the second imageplane and the projector plane; and the step of determining the deviationincludes determining that the position of the at least one point on oneof the at least two planes is different than an expected position. 5.The method of claim 4 wherein the determined deviation is inserted intoa field of error.
 6. The method according to claim 5 wherein the step ofdetermining the second calibration parameters is based on the type oferror field and the size of the deviations.
 7. The method of claim 1wherein the at least one point includes a first point and a secondpoint, the first point and second point being selected to be positionedat different distances from the 3D measuring device.
 8. The method ofclaim 1 wherein the projector, the first camera and the second cameraare positioned in a triangular arrangement.
 9. The method of claim 1wherein: the 3D measuring device further includes a two-dimensional (2D)camera; and the at least one point corresponds to a feature on theobject.
 10. The method of claim 9 further comprising recording a thirdimage with the 2D camera, the third image including the feature.
 11. Themethod of claim 10 wherein the step of determining the deviationincludes comparing the first image of the at least one point and thesecond image of the at least one point with the feature in the thirdimage.
 12. The method of claim 11 wherein the first image, the secondimage and the third image are recorded simultaneously.
 13. The method ofclaim 11 wherein the third image is acquired at second time, the secondtime being different than the first time.
 14. The method of claim 13wherein the step of emitting the light pattern is a pulsed pattern, thepattern being emitted during the first time and not emitted during thesecond time.
 15. The method of claim 12 wherein the 3D measuring devicefurther includes a filter arranged to block a wavelength of light, theprojector being configured to emit the light pattern at the wavelengthof light.
 16. The method of claim 15 wherein the wavelength of light isan infrared wavelength or an ultraviolet wavelength.
 17. A method foroptically scanning and measuring an environment, the method comprising:providing a three-dimensional (3D) measurement device having a firstcamera, a second camera and a projector, the 3D measurement devicefurther having a control and evaluation device operably coupled to theat least one camera and the projector, the control and evaluation devicehaving memory; emitting at a first wavelength a light pattern onto anobject with the projector; recording a first image of the light patternwith the first camera at a first time; recording a second image of thelight pattern with the second camera at the first time; producing a 3Dscan of the object based at least in part on the first image and thesecond image; selecting a first point in the center of the lightpattern, the first point being in the first image and the second image;selecting a second point in the light pattern, the second point beingpositioned away from the second of the light pattern; determining adeviation in the first wavelength based on the first point and thesecond point; and storing the deviation in memory.
 18. The method ofclaim 17 wherein the projector further includes a diffractive opticalelement.
 19. The method of claim 18 wherein the first point ispositioned at a zeroth order of diffraction and the second point islocated at a position of a higher order of diffraction.
 20. The methodof claim 17 further comprising: determining a second wavelength of lightbased at least in part on the deviation; and emitting at the secondwavelength of light the light pattern onto the object.
 21. The method ofclaim 17 wherein the projector, the first camera and the second cameraare positioned in a triangular arrangement.